add retries for ephemeral instance api calls #61

steffyP · 2026-01-27T13:46:45Z

As currently the API for ephemeral instances is known to be unstable, this PR aims to make to retry failing API calls and to capture additional logs in case of failure cases.
This way it will be easier to pinpoint issues in the future.

Example logs:

...
curl command failed while fetching instances. Response: {"error": true, "message": "compute.general_error"}
Command failed with exit code 1. Retrying in 5 seconds... (1/5)
Creating preview environment ...
Created preview environment with endpoint URL: https://ls-4ra30eqlofl88.sandbox.localstack.cloud/
...
Fetching logs for preview-localstack-setup-localstack-61 ...
Instance is not ready yet, waiting for 'Ready.' message in logs...
Command failed with exit code 1. Retrying in 5 seconds... (1/5)
preview-localstack-setup-localstack-61 logs:
Localstack extensions installer: installing extensions defined EXTENSIONS_AUTO_INSTALL
....

changes:

retry failing API call up to 5 times, with a 5 second waiting time in between (no guarantee this is enough, but it was sufficient for our CI and we have the logs for debugging issues)
log details (e.g. retry, the actual response)
added a helper script that will be loaded so that we can reuse the function
added additional check to make sure that logs are successfully retrieved

Suggested future work (once the API is stable):

rather use the localstack CLI than hardcoding the API calls, as any changes to the API will need changes to the action as well
- As @lukqw pointed out the CLI may not have the capabilities of handling ENVs that have been set (regarding autoloading of pods, extensions etc).
add more tests, e.g. some attributes are currently not tested or verified: auto-load-pod, ci-project

github-actions · 2026-01-27T13:47:26Z

The ephemeral instance for the application preview has been shut down

steffyP · 2026-01-29T17:09:50Z

.github/workflows/ephemeral.yml

+  pull_request:
+    paths-ignore:
+      - ./*.md
+      - LICENSE


removing the workflow_dispatch that I added in a previous PR, as it doesn't make sense with the logic that the ephemeral instances are currently following (e.g. searching for the PR number and adding comment to the PR)

steffyP · 2026-01-29T17:10:48Z

ephemeral/shutdown/action.yml

-    description: 'LocalStack API key used to access the platform api'
-    required: true
+    description: 'LocalStack Auth Token used to access the platform api'
+    required: false


the localstack-api-key is not documented, neither is it required or currently used in our own CI test. It is a fallback in case the LOCALSTACK_AUTH_TOKEN and LOCALSTACK_API_KEY are not set

Thanks, that's a good point! That's maybe something to revisit in the future. The term is outdated (we only have auth tokens now) and it seems a bit confusing to configure...

Just to add some more context here:
When someone tries to call the ephemeral instance API we require them to be authenticated.
Since the CI integration cannot provide email/password, we instead authenticate through an auth token.

The easiest way to do that is to use a ls-api-key auth header in combination with a auth token.
We could also switch to the Authorization: header in combination with an auth token, but on the backend we do not expect a plain text token in combination with that header. Happy to share more details here offline.

We could also rename the input variable and still rely on the ls-api-key header FWIW, since we have backwards compatibility.

following our offline discussion to have a paper-trail here:
there was a misunderstanding: we are not questioning the ls-api-key usage and we know we need to provide a auth-token/key for the request.

The initial comment is really just about the input parameter localstack-api-key for this workflow, that is used for setting the ls-api-key but only as a "last resort fallback" e.g. when the ENVs are not set.
e.g.

AUTH_HEADER="ls-api-key: ${LOCALSTACK_AUTH_TOKEN:-${LOCALSTACK_API_KEY:-${{ inputs.localstack-api-key }}}}"

Therefore, I changed the input to required: false for the workflow.

ephemeral/startup/action.yml

alexrashed

Thanks a lot for fixing the usage of ephemeral instances by implementing the retry logic!
From my perspective this is looking great. I think there is potential to move this to the CLI and reuse it here in the future, but that would definitely be out of scope for this PR!

Given that @lukqw is the expert on ephemeral instances and also has contributed to this feature in the action in the past, I think it might be good to maybe also get a quick review from you as well? 🙏🏽

alexrashed · 2026-01-30T09:08:48Z

ephemeral/shutdown/action.yml

-    description: 'LocalStack API key used to access the platform api'
-    required: true
+    description: 'LocalStack Auth Token used to access the platform api'
+    required: false


Thanks, that's a good point! That's maybe something to revisit in the future. The term is outdated (we only have auth tokens now) and it seems a bit confusing to configure...

ephemeral/startup/action.yml

alexrashed · 2026-01-30T09:26:33Z

ephemeral/retry-function.sh

+# retry() function: Retries a given command up to 'retries' times with a 'wait' interval.
+# Usage: retry <command>
+# Example: retry my_api_call_function
+retry() {


@lukqw I think this is where your input would really be valuable. Is this retry mechanism what you would expect a client to do if they get a failure response from the ephemeral instance API?
This effectively would be a classic retry machanism, 5 retries, 5 seconds constant backoff.

Thanks for the ping! I believe the approach here would be good enough to bridge the problem we currently have in our backend integration. 👌

I suppose an even safer approach would be to roll with some exponential back-off mechanism, but that could also be overkill / not necessary at the current stage.

Thanks a lot for diving into this @steffyP! 🙏 🚀

lukqw

Sorry for the late review folks! I was wrapped up in other topics unfortunately 🦥

Praise: Thanks a lot @steffyP for providing such a neat abstract retry function we can use to make our CI integration with ephemeral instances stabler 💪 🚀

lukqw · 2026-02-04T16:08:14Z

ephemeral/retry-function.sh

+# retry() function: Retries a given command up to 'retries' times with a 'wait' interval.
+# Usage: retry <command>
+# Example: retry my_api_call_function
+retry() {


Thanks for the ping! I believe the approach here would be good enough to bridge the problem we currently have in our backend integration. 👌

I suppose an even safer approach would be to roll with some exponential back-off mechanism, but that could also be overkill / not necessary at the current stage.

Thanks a lot for diving into this @steffyP! 🙏 🚀

lukqw · 2026-02-04T16:13:23Z

ephemeral/shutdown/action.yml

-    description: 'LocalStack API key used to access the platform api'
-    required: true
+    description: 'LocalStack Auth Token used to access the platform api'
+    required: false


Just to add some more context here:
When someone tries to call the ephemeral instance API we require them to be authenticated.
Since the CI integration cannot provide email/password, we instead authenticate through an auth token.

The easiest way to do that is to use a ls-api-key auth header in combination with a auth token.
We could also switch to the Authorization: header in combination with an auth token, but on the backend we do not expect a plain text token in combination with that header. Happy to share more details here offline.

We could also rename the input variable and still rely on the ls-api-key header FWIW, since we have backwards compatibility.

test retry loops to create ephemeral instances

ff31d3c

steffyP added 11 commits January 27, 2026 14:53

fixup

3ca264f

fixup

dc0474e

fixup

363f287

set +e inside the retry loop

b7d3d58

add more debug messages

44a4b1e

retry retrieving logs

8d3353d

adding more debug information for logs

89ed704

refactored retry function into separate script

8f70f75

fixup

b3ad5bc

fixup

5f7b169

debug log result

3ba8a36

steffyP force-pushed the fix-ephemeral branch from 2224b31 to 3ba8a36 Compare January 29, 2026 15:54

steffyP added 3 commits January 29, 2026 17:18

more debug messages, fail if the response contains error key

0ff0b96

fixup

dba16e5

fixup

9036a17

steffyP changed the title ~~test retry loops to create ephemeral instances~~ add retries for ephemeral instance api calls Jan 29, 2026

fixup

e88acfc

steffyP commented Jan 29, 2026

View reviewed changes

ephemeral/startup/action.yml Show resolved Hide resolved

steffyP marked this pull request as ready for review January 29, 2026 17:34

steffyP requested a review from alexrashed January 29, 2026 17:34

alexrashed approved these changes Jan 30, 2026

View reviewed changes

alexrashed requested a review from lukqw January 30, 2026 09:28

steffyP mentioned this pull request Feb 4, 2026

update actions and use hashes instead of version tag #60

Open

lukqw approved these changes Feb 4, 2026

View reviewed changes

steffyP merged commit 7c4d408 into main Feb 4, 2026
6 checks passed

steffyP deleted the fix-ephemeral branch February 4, 2026 17:02

add retries for ephemeral instance api calls #61

add retries for ephemeral instance api calls #61

Conversation

steffyP commented Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alexrashed left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lukqw left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

steffyP commented Jan 27, 2026 •

edited

Loading

github-actions bot commented Jan 27, 2026 •

edited

Loading